Systems and methods for estimating 3d position and movement from tactile signals

ABSTRACT

Systems and methods are provided for estimating 3D poses of a subject based on tactile interactions with the ground. Test subject interactions with the ground are recorded using a sensor system along with reference information (e.g., synchronized video information) for use in correlating tactile information with specific 3D poses, e.g., by training a neural network based on the reference information. Then, tactile information received in response to a given subject interacting with the ground can be used to estimate the 3D pose of the given subject directly, i.e., without reference to corresponding reference information. Certain exemplary embodiments use a sensor system in the form of a pressure sensing carpet or mat, although other types of sensor systems using pressure or other sensors can be used in various alternative embodiments.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This patent application claims the benefit of U.S. Provisional PatentApplication No. 63/007,675, entitled “SYSTEMS AND METHODS FOR ENABLINGHUMAN ACTIVITY LEARNING BY MACHINE-KNITTED, WHOLE-GARMENT SENSINGWEARABLES,” and filed Apr. 9, 2020, which is hereby incorporated hereinby reference in its entirety.

FIELD

The present disclosure relates to manufacturing whole-garment sensingwearables, and more particularly provides for knitting techniques thatallow for automated processes to produce such wearable on a large scale.The whole-garment sensing wearables enable human activity learning notachievable by existing smart textiles.

BACKGROUND

Organisms in nature extract information and learn from the externalenvironment through constant physical interactions. As an example,humans leverage their powerful tactile sensory system (skin on hands,limbs, and torso) to perform complex tasks, including dexterous graspand locomotion. Humans interact with the external environment every daythrough rich tactile perception. This important sensing modalityremains, however, challenging for robots to replicate, as skin-likesensory interfaces are still highly limited in terms of performance,scalability, and cost. Monitoring and understanding of interactionsbetween humans and the physical world provide fundamental knowledge forhuman behavior study, and to improve health care, biomimetic robots,human-computer interactions, augmented virtual/virtual reality (AV/VR),and others. Whereas visual and audio-based datasets are commonly used totrack and analyze human-environment interactions, equivalent richtactile datasets are rare.

Recently, the coupling of tactile information and machine learning toolshas enabled the discovery of signatures of human grasping. However,recording and analysis of whole-body interactions are extremelychallenging due to the lack of inexpensive large-scale conformalwearable sensors that are compatible with human activities.

To the extent sensors or the like have been incorporated into textiles,such incorporation results in rigid to semi-rigid garments that areneither as comfortable or as functional as their counterpart garmentsthat are not “smart.” The weaving techniques utilized in most instancesresults in such rigid to semi-rigid garments. Further, to the extenttechniques such as embroidery are used to incorporate sensors to form“smart textiles,” while they may result in more comfortable andfunctional textiles, such techniques are not scalable. Thus, thesetechniques have limited value to possibly no value to companies tryingto produce “smart textiles” for any commercial purposes. Despitetremendous progress of wearable electronics benefiting from advancedmaterials, designs, and manufacturing techniques, automatedmanufacturing of conformal sensing textiles at whole-body scale withlow-cost materials has not been realized yet.

Accordingly, there is a need for wearable sensors that can bemass-produced at a low cost and that can be utilized to enable humanactivity learning. There is likewise a need to generate data sets fromthe use of such wearable sensors and to use that data to generate avariety of determinative and/or predictive outcomes, including but notlimited to determining present or future actions based on data sensed bywearable sensors. Still further, there is a need to better be able toinfer, predict, and/or determine a particular motion or activity basedon a limited amount of information or data.

SUMMARY

In accordance with one embodiment of the invention, a system foridentifying activity of a subject relative the ground comprises atactile sensing floor covering for sensing interaction of the subjectwith the ground and a processing system in communication with the sensorsystem. The processing system includes at least one processor coupled toa non-transitory memory containing instructions executable by the atleast one processor to cause the system to receive an input tactilesequence produced from sensor signals generated by the tactile sensingfloor covering sensor system; compare the received input tactilesequence against information in a database that correlates tactileinformation to particular activities; and identify the activity of thesubject based on the comparison.

In various alternative embodiments, the identified activity may includeat least one of an identified movement or an identified position of atleast one part of the subject. The instructions may further cause thesystem to trigger a notification based on the identified activity, suchas, for example, an alarm, a warning, and/or an indication of an earlydisease detection. The tactile sensing floor covering may include atleast one of a carpet, rug, mat, floor cloth, pad, plank, tile, sheet,or other flooring product. The tactile sensing floor covering mayinclude a piezoresistive pressure sensing matrix fabricated by aligninga network of orthogonal conductive threads as electrodes on each side ofa commercial piezoresistive film, wherein each sensor is located at theoverlap of orthogonal electrodes. The instructions may further cause thesystem to implement an encoder that maps the input tactile sequence intoa 2D feature map, expands and repeats the 2D feature map to transformthe 2D feature map into a 3D feature volume comprising a plurality ofvoxels, and appends an indexing volume indicating the height of eachvoxel, and to implement a decoder that runs the appended and indexed 3Dfeature volume through a set of decoding layers to generate a predictedconfidence map for each of a plurality of keypoints, wherein thepredicted confidence map is used for comparing the input tactilesequence against information in the database that correlates tactileinformation to particular activities and identifying the activity of thesubject based on the comparison. The processing system may include aneural information processing system. The instructions may further causethe system to collect tactile information for a plurality of testsubjects along with reference information and process the collectedtactile information and the reference information to produce theinformation in the database that correlates tactile information toparticular activities. The system may include at least one camera,wherein the reference information comprises video or images from the atleast one camera of the test subjects producing the collected tactileinformation.

In accordance with another embodiment of the invention, a method foridentifying activity of a subject relative the ground involvesreceiving, by a processing system, an input tactile sequence producedfrom sensor signals generated by a tactile sensing floor covering thatsenses interaction of the subject with the ground; comparing, by theprocessing system, the received input tactile sequence againstinformation in a database that correlates tactile information toparticular activities; and identifying, by the processing system, theactivity of the subject based on the comparison.

In various alternative embodiments, the identified activity may includeat least one of an identified movement or an identified position of atleast one part of the subject. The method may further includetriggering, by the processing system, a notification based on theidentified activity such as, for example, an alarm, a warning, and/or anindication of an early disease detection. The tactile sensing floorcovering may include at least one of a carpet, rug, mat, floor cloth,pad, plank, tile, sheet, or other flooring product. The tactile sensingfloor covering may include a piezoresistive pressure sensing matrixfabricated by aligning a network of orthogonal conductive threads aselectrodes on each side of a commercial piezoresistive film, whereineach sensor is located at the overlap of orthogonal electrodes. Themethod may further involve implementing, by the processing system, anencoder that maps the input tactile sequence into a 2D feature map,expands and repeats the 2D feature map to transform the 2D feature mapinto a 3D feature volume comprising a plurality of voxels, and append anindexing volume indicating the height of each voxel; and implementing,by the processing system, a decoder that runs the appended and indexed3D feature volume through a set of decoding layers to generate apredicted confidence map for each of a plurality of keypoints, whereinthe predicted confidence map is used for comparing the input tactilesequence against information in the database that correlates tactileinformation to particular activities and identifying the activity of thesubject based on the comparison. The processing system may include aneural information processing system. The method may further involvecollecting tactile information for a plurality of test subjects alongwith reference information and processing the collected tactileinformation and the reference information to produce the information inthe database that correlates tactile information to particularactivities. The reference information may include video or images of thetest subjects producing the collected tactile information.

The present disclosure also provides for a textile-based tactilelearning platform that allows researchers to record, monitor, and learnhuman activities as well as associated interactions with the physicalworld. The platform can be implemented as a system or method, employingnovel, functional (e.g., piezoresistive) fibers that are inexpensive(about US$0.2/m), in conjunction with industrial whole-garment machineknitting, which can be automated, and machine learning workflow,including new calibration and learning algorithms, for examplecomputational pipelines for human-environment interaction recording andlearning. The e-scalable manufacturing of this new platform isdemonstrated through several non-limiting examples of conformal sensingtextiles (over 1000 sensors), e.g., glove, sock, vest, robotic armsleeve. Further, the disclosed platform can perform weakly supervisedsensing correction, endowing strong adaptability to variations inresponse of individual sensing elements. The present disclosure hasresulted in creating a rich dataset (over a million frames) on diversehuman-environment interactions, which can be used, by way ofnon-limiting examples, to classify objects/activities, distinguishenvironments, predict whole-body poses, discover motion signatures,grasping, and locomotion. The disclosures provided for herein open upnew possibilities in wearable electronics, functional textiles, healthmonitoring, and robot manipulation, among other fields.

One exemplary embodiment of a textile of the present disclosure includesa plurality of functional fibers that are interconnected by loops formedfrom the plurality of functional fibers such that the plurality offunctional fibers forms a knit. The textile also includes a plurality ofsensors disposed throughout the textile. The sensors are formed by theplurality of functional fibers.

The functional fibers can include a conductive core and a piezoresistivecoating disposed around a circumference of the conductive core. Thecoating can cover an entire circumference of at least a portion of theconductive core. The conductive core can have many differentconfigurations and be made of a variety of materials. One material canbe used to form the core, or a plurality of different materials can beused to form the core. In some embodiments, the conductive core includesstainless steel. Likewise, the piezoresistive coating can have manydifferent configurations and be made of a variety of materials. Onematerial can be used to form the coating, or a plurality of differentmaterials can be used to form the coating. In some embodiments, thepiezoresistive coating can include a polydimethylsiloxane elastomer.

The textile can include, or otherwise be, a wearable garment. Somenon-limiting examples of wearable garments that can be the textileinclude a glove, a sock, a top, a bottom, headwear, or a sleeve.Wearable garments are by no means limited to clothes though, as othertextiles or garments that can be placed on and/or over an object, human,or animal can also be a wearable garment in the context of the presentdisclosure. The textile can be flexible.

The plurality of functional fibers can include at least one of automaticinlays or manual inlays and, in some such embodiments, the functionalfibers can include a combination of automatic and manual inlays. Theplurality of sensors can be configured to adapt to environmental changesand/or can be configured to restore from self-deficit. In someembodiments, the plurality of sensors can be configured to develop aself-supervised sensing pipeline that automatically calibrates aresponse of the individual sensor.

One exemplary method of manufacturing a textile of the presentdisclosure includes knitting a plurality of functional fibers togetherusing interconnected loops to form a textile having a plurality ofsensors disposed in the textile. The sensors are formed by the pluralityof functional fibers. As described herein, and as will be appreciated bya person skilled in the art in view of the present disclosures, theaction of knitting is significantly different than the actions ofweaving and/or embroidering. The present methods and systems areintended to not use weaving or embroidery techniques in the formation ofthe whole garments themselves.

In at least some embodiments, the action of knitting a plurality offunctional fibers together using interconnected loops can includeoperating an automated machine to perform the knitting. Knitting aplurality of functional fibers together using interconnected loops canalso include digitally knitting the plurality of functional fiberstogether using interconnected loops. In some embodiments, knitting aplurality of functional fibers together using interconnected loops caninclude forming at least one of automatic inlays or manual inlays withthe plurality of functional fibers. In some such embodiments, the fiberscan be formed using a combination of automatic and manual inlays.

The plurality of functional fibers can include a conductive core and apiezoresistive coating. The coating can be disposed around acircumference of the conductive core such that the coating covers anentire circumference of at least a portion of the conductive core. Asdiscussed above, a variety of materials can be used for the conductivecore and/or the coating, such materials being able to be used asstandalone materials or as a part of a blend or mixture. In someembodiments, the conductive core can include stainless steel and/or thepiezoresistive coating can include a polydimethylsiloxane elastomer.

As also discussed above, the textile can include, or otherwise be, awearable garment. Some non-limiting examples of wearable garments thatcan be the textile include a glove, a sock, a top, a bottom, headwear,or a sleeve. Wearable garments are by no means limited to clothesthough, as other textiles or garments that can be placed on and/or overan object, human, or animal can also be a wearable garment in thecontext of the present disclosure. The textile can be flexible.

One exemplary system for manufacturing a textile provided for in thepresent disclosure includes a knitting machine that is configured toknit a plurality of functional fibers together to form a textile usinginterconnected loops. The knitting machine is configured to operate inan automated manner.

In some embodiments, the system can include a fiber-feeding system. Thefiber-feeding system can include a transport system, a coating device,and a curing device. The transport system can be operable to advance aconductive core of a functional fiber of the plurality of functionalfibers. The coating device can be configured to apply a piezoresistivecoating to the conductive core advanced by the transport system suchthat the piezoresistive coating covers an entire circumference of atleast a portion of the conductive core. The curing device can beconfigured to cure the piezoresistive coating to the conductive core toform the functional fiber of the plurality of functional fibers.

The knitting machine can be configured to form one or both of automaticinlays or manual inlays with the plurality of functional fibers knittedtogether using interconnected loops. In some embodiments, the knittingmachine is configured to digitally knit the plurality of functionalfibers together using interconnected loops.

Similar to other exemplary embodiments described above, the textileformed by the knitting machine can include, or otherwise be, a wearablegarment. Some non-limiting examples of wearable garments that can be thetextile include a glove, a sock, a top, a bottom, headwear, or a sleeve.Again, wearable garments are by no means limited to clothes though, asother textiles or garments that can be placed on and/or over an object,human, or animal can also be a wearable garment in the context of thepresent disclosure. Additionally, the textile formed by the knittingmachine can be flexible.

One exemplary embodiment of a fiber for use in a textile as provided forin the present disclosure includes a conductive core and apiezoresistive coating. The piezoresistive coating is disposed around acircumference of the conductive core such that the coating covers anentire circumference of at least a portion of the conductive core. Asdiscussed above, a variety of materials can be used for the conductivecore and/or the coating, such materials being able to be used asstandalone materials or as a part of a blend or mixture. In someembodiments, the conductive core can include stainless steel and/or thepiezoresistive coating can include a polydimethylsiloxane elastomer.

The present disclosure also provides for an exemplary method forcalibrating sensors associated with a textile. The method includesreceiving a plurality of readings from a plurality of sensors associatedwith a textile that results from an action being performed that yieldsthe plurality of readings. The readings are indicative of one or moreparameters used to identify an activity. The method also includesrecording the plurality of readings and synchronizing the plurality ofreadings to calibrate the plurality of sensors.

In some embodiments, the plurality of readings can include readings ofat least one of: a pressure, a temperature, a pH level, a chemicallevel, an electro-magnetic property, an acoustic parameter, or avibration. Other parameters can also be measured or otherwise read. Ininstances where the readings are a pressure, performing an action thatyields the plurality of readings can include pressing the textileagainst a digital scale a plurality of times, with the plurality ofreadings being from each time the textile is pressed against the digitalscale. Each sensor of the plurality of sensors can be calibratedindividually. Further, each calibration for each sensor can be stored inconjunction with the respective sensor of the plurality of sensors.

One exemplary method of training a neural network in view of the presentdisclosures includes providing a small sequence of unprocessed tactileresponses to a neural network and causing the neural network to output asingle frame with the same spatial resolution as the small sequence ofunprocessed tactile responses.

The method can also include optimizing the neural network via stochasticgradient descent on a plurality of objective functions. In someembodiments, the method can include increasing a correlation betweentactile response and the single frame outputted by the neural network.

A method for identifying an activity, such as a human activity, isanother exemplary method provided for in the present disclosure. Themethod includes receiving tactile information from a smart textile,comparing the tactile information against a database of tactileinformation that correlates the data to particular activities (e.g.,human activities), and identifying the activity based on the comparison.

In some embodiments, the activity is a human activity and it includesvarious actions related to movement. The identified human activity caninclude, for example, an identified movement and/or identified positionof body parts of a human. The method can also include triggering anotification in view of the identified activity. For example, thenotification can include at least one of an alarm, a warning, or anindication of an early disease detection.

The present disclosure also provides for one or more systems that areable to perform one or more of the methods described above or otherwisedescribed herein.

BRIEF DESCRIPTION OF DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

Those skilled in the art should more fully appreciate advantages ofvarious embodiments of the invention from the following “Description ofIllustrative Embodiments,” discussed with reference to the drawings, inwhich:

FIG. 1A is a photograph showing a low-cost, high-density, large-scaleintelligent carpet system to capture the real-time human-floor tactileinteractions, in accordance with certain exemplary embodiments;

FIG. 1B shows inferred 3D human poses from the captured tactileinteractions of a person at various stages when standing up from asitting position, including, for each stage, an RGB image captured bythe camera (top), a pressure map representing signals received from thecarpet system (middle), and a reconstructed 3D skeleton produced fromthe RGB image (bottom), in accordance with certain exemplaryembodiments;

FIG. 2A is a schematic diagram showing relevant components of thetactile data acquisition hardware as used in the prototype system;

FIG. 2B shows typical pressure maps captured by the carpet from diversehuman poses and activities, in accordance with the tactile dataacquisition hardware of FIG. 2A;

FIG. 3 is a schematic diagram showing 3D keypoint confidence mapgeneration in accordance with the prototype system;

FIG. 4 is a schematic diagram showing an overview of the model for 3Dhuman pose estimation;

FIG. 5 shows Euclidean (L2) distance between the predicted single-person3D skeleton (21 keypoints) and the ground truth label;

FIGS. 6A-6D show some qualitative results of single-person 3D human poseestimations across time steps;

FIG. 7 shows model performance with different sensing resolutions andnumber of input frames;

FIGS. 8A-8C show generalization results including localization error ofpredictions on seen tasks and individuals where the training wasperformed on the full dataset including all tasks and individuals (top),localization error of predictions on unseen tasks and individuals wherethe training was performed on a split dataset excluding specific actionsand individuals (middle) and qualitative results on unseen tasks andindividuals (bottom) for three tasks, specifically a lunge (FIG. 8A), apush-up (FIG. 8B), and a sit-up (FIG. 8C);

FIG. 9 shows results on action classification including a confusionmatrix of action classification using a linear classifier on the learnedfeatures from the pose estimation model and representative tactileframes from different actions;

FIG. 10 shows results on multi-person scenarios including Euclideandistance between the predicted multi-person 3D skeleton and the groundtruth;

FIGS. 11A-11B show some qualitative results of exemplary multi-person 3Dhuman pose estimation, where the images in FIG. 11B are a continuationof the sequence shown in FIG. 11A;

FIG. 12 shows typical failure cases encountered in the prototype system;

FIG. 13 shows exemplary embodiments of coaxial piezoresistive fibers,where FIG. 13A is a perspective view of exemplary embodiments of coaxialpiezoresistive fibers, the fiber being disposed on a roll andincorporated into garments; FIG. 13B is an optical microscope image ofeach of a stainless steel thread for use in conjunction with a coaxialpiezoresistive fiber, a functional fiber, and acrylic yarn; FIG. 13C isa scanning electron microscope (SEM) image of a cross-section asidentified in the functional fiber of FIG. 13B; FIG. 13D is an SEM imageof the functional fiber of FIG. 13C taken at a closer range; FIG. 13E isan SEM image of the functional fiber of FIG. 13D taken at a closer rangeand after shear mixing the functional fiber at a high rate of speed;FIG. 13F is a graph illustrating a change in resistance in response toload from a normal force associated with an exemplary embodiment of asensing wearable of the present disclosure; FIG. 13G is a graphillustrating sensor resistance over time associated with an exemplaryembodiment of a sensing wearable of the present disclosure; and FIG. 13His a graph illustrating sensor resistance in response to load associatedwith different combination of fabric structures;

FIG. 14 shows scalable manufacturing of machine knitted sensingwearables such as for data collection and learning, where FIG. 14A is aschematic illustration of one exemplary embodiment of a method ofmanufacturing a sensing wearable; FIG. 14B is a top perspective view ofone exemplary embodiment of a sensing wearable, as shown a pair ofgloves; FIG. 14C is a side perspective view of another exemplaryembodiment of a sensing wearable, as shown a sock; FIG. 14D is a sideperspective view of yet another exemplary embodiment of a sensingwearable, as shown a vest; FIG. 14E is a side perspective view of stillanother exemplary embodiment of a sensing wearable, as shown a sleevefor use with a robotic arm; and FIG. 14F is a schematic illustration ofvarious ways by which sensing wearables, such as those illustrated inFIGS. 14B-14E, can used to collect data and learn from the same;

FIG. 15 shows various forms of collecting tactile information usingsensing wearables, where FIG. 15A illustrates tactile informationcollected from pressing a sensing wearable glove on a digital scale;FIG. 15B illustrates tactile information collected by pressing a sensingwearable glove on a sensing wearable vest; FIG. 15C illustrates both araw signal image and a self-supervised image that results for each ofthree different objects when pressed against the sensing wearable gloveof FIG. 15A; FIG. 15D illustrates both a raw signal image and aself-supervised image that results for each of three positions when thesensing wearable vest of FIG. 15B contacts an object; FIG. 15Eillustrates both a raw signal image and a self-supervised image thatresults for each of two positions of a foot wearing a sensing wearablesock; and FIG. 15F illustrates both a raw signal image and aself-supervised image that results for each of two positions of arobotic arm wearing a sensing wearable sleeve;

FIG. 16 shows some exemplary interpretations of tactile information,where FIG. 16A illustrates example photographs and tactile frames thatcan assist in the identification of diverse sets of signatures; FIG. 16Billustrates a T-distributed Stochastic Neighbor Embedding (T-SNE) plotfrom a pose dataset from a sensing wearable vest; FIG. 16C illustratesexample photographs and tactile frames of letters pressed on a sensingwearable vest to classify sensory information, such as the letter andorientation; FIG. 16D illustrates a plot of effective input resolutionagainst classification accuracy, as well as a confusion matrix; FIG. 16Eis a schematic illustration of a human body illustrating 10 differentjoint angles that can be predicted by exemplary embodiments of thesystems and methods provided for herein; FIG. 16F is a graphillustrating the mean-squared error (MSE) in pose production; FIG. 16Gis a graph illustrating the MSE in effective input resolution of asensor of exemplary embodiments of the systems and methods provided forherein; FIG. 16H is a graph illustrating the MSE in number of inputframes (context window) of the systems and methods provided for herein;FIG. 16I provides a comparison of poses predicted by tactile footprintdata illustrated in the figure and actual poses associated with thegenerated tactile footprint data; FIG. 16J provides a comparison of timeseries predictions of poses from walking predicted by tactile footprintdata illustrated in the figure and actual poses from walking associatedwith the generated tactile footprint data; and FIG. 16K providesprincipal component analysis (PCA) on tactile maps from walking, withinsets therein corresponding to relevant tactile frames from thewalking;

FIG. 17A illustrates one exemplary embodiment of calibrating a sensingwearable glove using a digital scale;

FIG. 17B illustrates one exemplary embodiment of using tactile feedbackfrom sensing wearable socks to predict a stance of a wearer of the socksbased on the tactile feedback;

FIG. 17C illustrates another exemplary embodiment of using tactilefeedback from sensing wearable socks to predict a stance of a wearer ofthe socks based on the tactile feedback;

FIG. 17D illustrates one exemplary embodiment of calibrating a sensingvest using a calibrated sensing wearable glove;

FIG. 17E illustrates one exemplary embodiment of calibrating a sensingsleeve using a calibrated sensing wearable glove; and

FIG. 18 illustrates example photographs and tactile frames of oneexemplary embodiment of a sensing sleeve disposed on a robotic armreceiving real-time tactile feedback.

DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

Certain exemplary embodiments will now be described to provide anoverall understanding of the principles of the structure, function,manufacture, and use of the devices and methods disclosed herein. One ormore examples of these embodiments are illustrated in the accompanyingdrawings. The present disclosure is inclusive of U.S. Provisional PatentApplication No. 63/007,675, entitled “SYSTEMS AND METHODS FOR ENABLINGHUMAN ACTIVITY LEARNING BY MACHINE-KNITTED, WHOLE-GARMENT SENSINGWEARABLES,” and filed Apr. 9, 2020, including the Appendices appurtenantthereto, which was incorporated by reference above in its entirety andis referred to herein as “the priority patent application.” Anyreference to “the present disclosure,” “herein,” or similar statementsis inclusive of the accompanying drawings and the priority patentapplication including the Appendices, and references to Appendix A,Appendix B, or the Appendices refer specifically to the Appendices inthe priority patent application. Applicant expressly reserves the rightto amend this patent application to physically incorporate any of thesubject matter of the priority patent application, including any figuresin the Appendices.

Those skilled in the art will understand that the devices and methodsspecifically described herein and illustrated in the accompanyingdrawings are non-limiting exemplary embodiments and that the scope ofthe present disclosure is defined solely by the claims. The featuresillustrated or described in connection with one exemplary embodiment maybe combined with the features of other embodiments. Such modificationsand variations are intended to be included within the scope of thepresent disclosure.

Certain exemplary embodiments include systems and methods for estimating3D poses of a subject from tactile interactions with the ground byrecording the interactions with the ground using a tactile sensing floorcovering (e.g., in the form of a carpet, rug, mat, floor cloth, pad,plank, tile, flooring product, etc., although it should be noted thatsuch tactile sensing devices are not limited to placement on the floorand instead can be placed on virtually any surface such as walls, doors,furniture, machinery, etc. for sensing subject-to-ground orsubject-to-surface interactions, including non-flat surfaces that can becovered by flexible floor coverings or by otherwise altering a floorcovering to comply with the contours of the surface) incorporating asensor system and processing the sensor signals from the incorporatedsensor system (which essentially provide a 2D mapping of theinteractions with the ground) into estimated 3D poses. Such 3D poseestimation can be useful in a wide range of disciplines including,without limitation, action recognition, gaming, healthcare, androbotics. Also, as opposed to 3D pose estimation using images or video,which can present privacy concerns and also do not perform well in thepresence of occlusions, 3D pose estimation based on tactile interactionswith the ground can be done more securely and do not suffer from “lineof sight” issues. For purposes of this discussion and claims, the term“ground” is used generically to refer to a substantially fixed surfaceon which the subject is supported such as for standing or walking (e.g.,a floor, or perhaps a table or other surface such as for a machine orrobot), and terms such as “ground” and “floor” may be usedinterchangeably.

Aspects are described with reference to an implemented prototype systemconfigured for estimating 3D poses of human subjects based on pressurereadings from a tactile sensing floor covering in the form of a carpetincorporating a pressure sensor system (which may be referred to hereinfor convenience as an “intelligent carpet”), although it should be notedthat other forms of tactile sensing floor coverings (e.g., rug, mat,floor cloth, pad, plank, tile, sheet, or other flooring product)incorporating pressure and/or other types of sensors (e.g., temperature,pH, chemical, electromagnetic, electrodermal, acoustic, vibration, etc.)may be used in various alternative embodiments (where, for purposes ofthis discussion and claims, all such sensors are deemed to providetactile information when produced due to a subject's physicalinteraction with the ground). Further, the same or similar systems andmethods can be used to estimate position and movement of other subjectsthat interact with the ground including, without limitation, animals andeven non-living subjects such as machinery or robots. Thus, for exampleand without limitation, a tactile sensing floor covering can be placedon top of another flooring layer (e.g., carpet, rug, or mat on top of anexisting floor), under another flooring layer (e.g., a pad under acarpet or rug), or as a top flooring layer (e.g., sensors integratedinto flooring planks, tiles, etc.).

The following is a description of the hardware setup for tactile dataacquisition, pipeline for ground truth 3D keypoint confidence mapgeneration, as well as data augmentation and synthesis for multi-personpose estimation, in accordance with the prototype system.

FIG. 1A is a photograph showing the data collection setup for theprototype system including a low-cost, high-density, large-scaleintelligent carpet 10 to capture the real-time human-floor tactileinteractions. Also shown in this figure is a camera 12 used to captureimages or video of the subject synchronized with readings from thecarpet 10.

FIG. 1B shows inferred 3D human poses from the captured tactileinteractions of a person at various stages when standing up from asitting position, including, for each stage, red green blue (RGB) imagecaptured by the camera (top), a pressure map representing signalsreceived from the carpet system (middle), and a reconstructed 3Dskeleton produced from the RGB image (bottom) with different body partshighlighted using different colors (e.g., green for legs, blue for feet,red for torso, etc.), in accordance with certain exemplary embodiments.Each of these will be described in greater detail below.

FIG. 2A is a schematic diagram showing relevant components of thetactile data acquisition hardware as used in the prototype systemincluding a tactile sensing carpet 10 approximately 6 ft×6 ft square(i.e., spanning around 36 ft²) incorporating 9,216 sensors with aspacing of about 0.375 inches that can be seamlessly embedded on thefloor and the corresponding readout (RO) circuits 15 that capture thesensor signals, multiplexing (MUX) circuit 16 that processes thecaptured sensor signals and provides the processed sensor signals to theprocessing system 20, and two cameras 12 that enable real-timerecordings of high-resolution human-ground tactile interactions for useby a processing system 20 such as a neural information processingsystem. The tactile sensing carpet 10 of the prototype system wascomposed of a piezoresistive pressure sensing matrix fabricated byaligning a network of orthogonal conductive threads as electrodes oneach side of the commercial piezoresistive films. Each sensor locates atthe overlap of orthogonal electrodes and can measure pressure up toabout 14 kPa with the highest sensitivity of about 0.3 kPa. This tactilesensing carpet is low-cost (˜$100), easy to fabricate, and robust forlarge-scale data collection. Using the prototype system as depicted inFIG. 2A, the tactile frames with 9,216 individual sensing readouts canbe collected, by way of non-limiting example, at a rate of about 14 Hz.With such a large-scale high-resolution tactile sensing platform, theprototype system can not only capture people's foot pressure maps, butalso can capture the full tactile interactions between the human and thefloor when people are performing complex activities. It should be notedthat the configuration of this exemplary tactile sensing carpet can beused to form sensor systems for other embodiments such as wearablesensor systems of the types described below. It also should be notedthat tactile sensing floor coverings are not limited to this type oftactile sensing carpet but instead virtually any pressure sensingcarpet, rug, or other pressure sensing system can be used in variousalternative embodiments. For example, similar tactile sensing carpetscan be woven or otherwise produced from coaxial piezoresistivefunctional fibers of the types discussed below, and exemplary carpetsformed from such coaxial piezoresistive functional fibers are discussedbelow.

With this hardware, over 1,800,000 synchronized tactile and visualframes were collected for 10 different individuals performing a diverseset of daily activities, e.g., lying, walking, and exercising. Employingthe visual information as reference, a processing system comprising adeep neural network was implemented to infer the corresponding 3D humanpose using only the tactile information. Resulting from thisimplementation is a database that correlates tactile information toparticular human activities such as, for example, standing, sitting,transitioning from sitting to standing or vice versa, movements of thebody, or other activities. The processing system then can comparetactile information received from a sensor system to the information inthe database in order to identify an activity of the human based on thecomparison. For example, the identified activity can include anidentified movement or an identified position of at least one body part.The prototype system was found to predict the 3D human pose with averagelocalization error of less than about 10 cm compared with the groundtruth pose obtained from the visual information. The learnedrepresentations from the pose estimation model, when combined with asimple linear classifier, allowed performance of action classificationwith an accuracy of about 98.7%. Included below are ablation studies andan evaluation of how well the model generalized to unseen individualsand unseen actions. Moreover, it is shown below that the prototypesystem can be scaled up for multi-person 3D pose estimation. Leveragingthe tactile sensing modality, embodiments open up opportunities forhuman pose estimation that is unaffected by visual obstructions in aseamless and confidential manner.

FIG. 2B shows typical pressure maps captured by the carpet from diversehuman poses and activities, in accordance with the tactile dataacquisition hardware of FIG. 2A. Specifically, the carpet captures thefeet pressure maps when people perform activities in upright positions,as well as the physical contacts between the human body (e.g., hands,limbs) and the floor when people perform exercises and complex actions(e.g., push-ups, sit-ups, and rolling).

The prototype system predicts 3D pose from only the tactile signals,which does not require any visual data and is fundamentally differentfrom past work in computer vision known to the inventors. The introducedtactile carpet has a lower spatial resolution than typical cameras.However, it essentially functions as a type of camera viewing humansfrom the bottom up. This type of data stream does not suffer fromocclusion problems that are typical for camera systems. Furthermore, itprovides additional information, such as whether humans are in contactwith the ground and the pressure they exert.

The prototype system implements 3D pose label generation as a pipelineto capture and generate the training pairs, i.e., synchronized tactileframes and 3D keypoint confidence maps. The system captures visual datawith two cameras that were synchronized and calibrated with respect tothe global coordinate of the tactile sensing carpet using standardstereo camera calibration techniques. In order to annotate the groundtruth human pose in a scalable manner, the system included astate-of-the-art vision-based system, OpenPose (e.g., Zhe Cao, TomasSimon, Shih-En Wei, and Yaser Sheikh. Realtime Multi-Person 2D PoseEstimation using Part Affinity Fields. In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPR), pages7291-7299, 2017; Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, andYaser Sheikh. OpenPose: Realtime Multi-Person 2D Pose Estimation usingPart Affinity Fields. arXiv preprint arXiv:1812.08008, 2018, each ofwhich is hereby incorporated herein by reference in its entirety), togenerate 2D skeletons from the images captured by the cameras.

Once the 2D skeletons are generated from the calibrated camera system,the system can triangulate the keypoints to generate the corresponding3D skeletons. The triangulation results may not be perfect in someframes due to perception noise or misdetection. To resolve this issue,the system can add a post-optimization stage to constrain the length ofeach link. More specifically, the system can first calculate the lengthof the links in the skeleton using the median value across the naivelytriangulated result for each person. For each specific person, thelength of the i^(th) link can be denoted as K_(i). The terms p^(A) andp^(B) can then be used to represent the detected N keypoints at aspecific time step from the two cameras, which lie in a 2D space, wherep^(A)={p₁ ^(A) . . . p_(N) ^(A)} and p_(k) ^(A)=(x_(k) ^(A),y_(k) ^(A)).The system can then calculate the length of each link from the naivetriangulation result and then can optimize the 3D location of thekeypoints p by minimizing the following loss function using stochasticgradient descent:

$\begin{matrix}{\mathcal{L}^{skeleton} = {{\sum\limits_{k = 1}^{N}{{{P^{A}p_{k}} - p_{k}^{A}}}} + {\sum\limits_{k = 1}^{N}{{{P^{B}p_{k}} - p_{k}^{B}}}} + {\sum\limits_{i = 1}^{N - 1}{{{\hat{K}}_{i} - K_{i}}}}}} & (1)\end{matrix}$

where there are N keypoints and N−1 links, p={p₁, . . . ,p_(N)} lie in3D space spanned by the world coordinate, p_(k)=(x_(k), y_(k), z_(k)).P^(A) and P^(B) are the camera matrices that project the 3D keypointsonto the 2D image frame. N=21 was used in the prototype system. Giventhe optimized 3D positions of the 21 keypoints on the human skeleton,the system can generate 3D keypoint confidence maps by applying a 3DGaussian filter over the keypoint locations on a voxelized 3D space.FIG. 3 is a schematic diagram showing 3D keypoint confidence mapgeneration in accordance with the prototype system, where the groundtruth voxelized 3D keypoint confidence maps are annotated by firstextracting 2D skeleton keypoints from RGB images using OpenPose, andthen 3D keypoints can be generated through triangulation andoptimization, and finally a 3D Gaussian filter can be applied.

When projecting the human skeletons to the x-y plane (FIG. 1), there isa spatial correspondence between the projection and the tactile signals,which allows for augmenting the dataset by rotating and shifting thetactile frames and the corresponding human skeletons. Due to therestriction of social distancing and the size of the sensing carpet,data collection was conducted with only one person at a time. Amulti-person dataset was synthesized by combining multiple single-personclips. In other words, the synchronized tactile frames and the generated3D keypoint confidence maps were added up from different recordingclips. For the sake of the prototype system, it was assumed that peoplerarely perform actions with one on top of the other, so it was assumedthat the pressure maps induced by the actions of different people willnot overlap at any given time. Therefore, the location of each personwas specified by creating anchor boxes of the human skeleton projectedonto the floor plane, and then frames with the Intersection over Union(IoU) larger than 0.1 were removed to ensure that the skeletons andtactile signals from different people did not overlap with each other.The training of the models was entirely based on the single-persondataset and the synthetic multi-person variants. Synchronized visual andtactile data were recorded for multiple people but only for evaluationpurposes.

The following presents details of the pose estimation model inaccordance with the prototype system including how the tactile frameswere transformed into 3D volumes indicating the confidence map of thekeypoints and how it was extended to multi-person scenarios.Implementation details are also presented.

For keypoint detection using tactile signals, the goal of the model isto take the tactile frames as input and predict the corresponding 3Dhuman pose. The ground truth human pose estimated from the multi-camerasetup is used as the supervision and to train the model to predict the3D confidence map of each of 21 keypoints, including head (nose), neck,shoulders, elbows, waists, hips (left, right and middle), knees, ankles,heels, small toes, and big toes. To include more contextual informationand reduce the effects caused by the sensing noise, instead of taking asingle tactile frame as input, the model takes a sequence of tactileframes spanning a temporary window of length M as input (FIG. 4). Foreach input segment, the model can process the spatiotemporal tactileinformation and output the keypoint confidence maps in 3D thatcorrespond to the middle frame. As shown in FIG. 1, the input tactileframes lie in 2D space, which has good spatial correspondence with thehuman skeleton over the x-y plane (the floor plane). The model can buildon top of a fully convolutional neural network to exploit such spatialequivariance. The encoder of the model can use 2D convolution to processthe tactile frames. Then, to regress the keypoints in 3D, the featuremap can be expanded by repeating it along a new dimension in the middleof the network (FIG. 4), which essentially transforms the 2D feature mapinto a 3D feature volume. However, naive 2D to 3D expanding viarepetition can introduce ambiguities as subsequent convolutional layersuse shared kernels to process the feature, particularly because it canbe difficult or impossible to determine the height of a specific voxel,making it hard to regress the keypoint location along the z-axis. Toresolve this issue, a new channel can be added to the 3D feature mapwith a 3-dimensional indexing volume, indicating the height of eachvoxel. Then, 3D convolution can be used to process the feature andpredict the 3D keypoint confidence map for each of the 21 keypoints. Thedetailed architecture and the size of the feature maps along theforwarding pass are shown in FIG. 4. The model can be optimized, forexample, by minimizing the Mean Squared Error (MSE) between thepredicted keypoint heatmap and the ground truth using, for example, anAdam optimizer (e.g., Diederik P Kingma and Jimmy Ba. Adam: A method forstochastic optimization. arXiv preprint arXiv:1412.6980, 2014, which ishereby incorporated herein by reference in its entirety). Spatialsoftmax can be used to transform the heatmap into the keypoint locationand to include an additional loss term

^(link) to constrain the length of each link in the skeleton to lie inthe range of normal human limb length. For each data point, the lossfunction can be defined as:

$\begin{matrix}{{\mathcal{L} = {{\frac{1}{N}{\sum\limits_{i = 1}^{N}{{H_{i} - {\hat{H}}_{i}}}}} + {\frac{1}{N - 1}{\sum\limits_{i = 1}^{N - 1}\mathcal{L}_{i}^{link}}}}},} & (2)\end{matrix}$

where N denotes the number of keypoints, N−1 is the number of links inthe skeleton, H_(i) and Ĥ_(i) represent the ground truth and thepredicted 3D keypoint confidence maps. The link loss can be defined asfollows:

$\begin{matrix}{\mathcal{L}_{i}^{link} = \left\{ \begin{matrix}{{K_{i}^{\min} - {\hat{K}}_{i}},} & {{{if}\mspace{14mu}{\hat{K}}_{i}} < {K_{i}^{\min}.}} \\{{{\hat{K}}_{i} - K_{i}^{\max}},} & {{{if}\mspace{14mu}{\hat{K}}_{i}} > {K_{i}^{\max}.}} \\{0,} & {{otherwise},}\end{matrix} \right.} & (3)\end{matrix}$

where {circumflex over (K)}_(i) is the link length calculated from theprediction, K_(i) ^(min) and K_(i) ^(max) represent the 3^(rd) and97^(th) percentile of each of the body limb length in the trainingdataset.

When moving into multi-person scenarios, each keypoint confidence mapcan contain multiple regions with high confidence that belong todifferent people. Therefore, the system can threshold the keypointconfidence map to segment out each of these high confidence regions, andthen can calculate the centroid of each region to transform it into the3D keypoint location. To associate the keypoints that belong to the sameperson, the system can start from the keypoint of the head and traversethrough the person's skeleton (represented as a tree) to include theremaining keypoints. Every time the system wants to add a new keypointto the person, e.g., the neck, the system can select the one amongmultiple extracted keypoint candidates with the closest L2 distance toits parent, e.g., head, which can have already been added to the personon the skeleton tree. This method works well when people are kept at acertain distance from each other, as well as possibly in other contexts.The inventors contemplate implementing more complicated and effectivetechniques to handle cases where people are very close to each other butwere unable to do so at the time of invention due to certain medicalprotocol issues (i.e., COVID-19 related).

The prototype system can be implemented using PyTorch (e.g., AdamPaszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, GregoryChanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, etal. Pytorch: An imperative style, high-performance deep learninglibrary. In Advances in neural information processing systems, pages8026-8037, 2019, which is hereby incorporated herein by reference in itsentirety). The model includes an encoder and a decoder. The encoder canmap the input tactile sequence into, for example, a 10×10 featurethrough 7 blocks of Conv2D-LeakyReLU-BatchNorm and then can expand andrepeat the feature along the last dimension to transform the 2D featuremap into a 3D feature volume. After appending an indexing volumeindicating the height of each voxel, the system can run the featurethrough a set of decoding layers to generate the predicted confidencemap for each keypoint. In the prototype system, the model can be trainedby minimizing Eq. 2 using a learning rate of 1e-4 and a batch size of32. FIG. 4 includes a schematic diagram showing an overview of the modelfor 3D human pose estimation. As shown in FIG. 4, the encoding part ofthe network can include seven (7) groups of layers. The Conv2D in thefirst five (5) and the 7^(th) layers use 3×3 kernels and 1×1 padding.The 6^(th) uses 5×5 kernels and zero padding. As shown, a 2×2 MaxPool2Dcan also be applied in the 2^(nd), 4^(th), and 7^(th) layers to reducethe resolution of the feature maps. The tactile feature maps can beexpanded to 3D, for example, by repeating the tensor nine (9) timesalong the last dimension, and then appending the channel with a 3Dindexing volume indicating the height of each voxel. The decodingnetwork can take in the resulting tensor and predict the 3D confidencemaps of the keypoints. The decoder can include, for example, five (5)layers of 3×3×3 3D convolution with a padding of 1×1×1. The 11^(th)layer can use, for example, a kernel size of 2×2×2 with a stride of two(2) to increase the resolution. Batch normalization and leaky rectifiedlinear unit (ReLU) can be applied after each layer, although in theillustrated embodiment, it is not applied after the last one, whereinstead a Sigmoid activation function is used to regress the confidencevalue.

In accordance with the present disclosure, single-person pose estimationwas trained with 135,000 pairs of tactile and visual frames andvalidated on 30,000 pairs of frames. Performance was tested on aheld-out test set with 30,000 tactile frames. Euclidean distance (L2)was used as the evaluation metric to compare the predicted 3D human poseto the corresponding ground truth human pose retrieved from the visualdata. FIG. 5 shows the Euclidean (L2) distance between the predictedsingle-person 3D skeleton (21 keypoints) and the ground truth label,i.e., the Euclidean (L2) distance of each keypoint and the localizationerror of each body part. The following table shows the average keypointlocalization error of body parts along the X, Y, and Z axis in thereal-world coordinate:

Shoul- El- An- Axis Ave. Head der bow Waist Hip Knee kle Feet X 6.8 6.46.3 8.9 10.9 4.6 5.8 5.6 6.4 Y 7.2 8.0 6.5 8.8 10.9 5.2 5.8 5.7 6.7 Z6.8 9.6 7.0 8.9 14.4 4.0 4.0 3.1 3.5

As shown, since the changes in pressure maps are dominated by thepositions and movements of the lower body and the torso, theirpredictions are more accurate. Thus, generally, keypoints on the lowerbody (e.g., knee and ankle) and the torso (e.g., shoulder and hip) holdhigher accuracy compared with the keypoints on the upper body (e.g.,waist and head). Further, the model can obtain better predictions if thekeypoints are closer to the torso on the skeleton tree—the predictionerror increases as the keypoints move further away from the torso, e.g.,shoulders to elbows, and then to the waist.

FIGS. 6A-6D show some qualitative results of single-person 3D human poseestimations across time steps, including, for each sequence, from top tobottom, the RGB image as ground truth annotation (only used here forvisualization purpose), the captured tactile frame, the ground truth 3Dskeleton, and the predicted 3D skeleton from the model using only thetactile frames (unit: cm). FIG. 6A depicts a person performing apush-up. FIG. 6B depicts a person performing a sit-up. FIG. 6C depicts aperson making mainly head movements. FIG. 6D depicts a person makingcertain body movements. The predicted poses are consistent over timewith a smooth transition along the corresponding trajectories. Ablationstudies were performed on the sensing resolution of the intelligentcarpet. To ablate the tactile sensing resolution, the value in each 2×2grid was reassigned with the average of the four values, which reducesthe effective resolution from 96×96 to 48×48, and then used the sametraining pipeline to derive the predictions. A similar procedure wasemployed for evaluating model performance with effective sensingresolutions of 24×24 and 12×12. FIG. 7 shows model performance withdifferent sensing resolutions (left) and number of input frames (right).As FIG. 7 illustrates, the prediction accuracy decreases with thedecrease of sensing resolution, which highlights the importance of ahigh density, large-scale tactile sensing platform. Based on an ablationstudy on the number of input frames, the best performance in this modelwas obtained with 20 input frames (˜1.5 sec).

Also, an evaluation of how well the model generalizes to unseenindividuals and activities was conducted. FIGS. 8A-8C showgeneralization results including localization error of predictions onseen tasks and individuals where the training was performed on the fulldataset including all tasks and individuals (top), localization error ofpredictions on unseen tasks and individuals where the training wasperformed on a split dataset excluding specific actions and individuals(middle) and qualitative results on unseen tasks and individuals(bottom) for three tasks, specifically a lunge (FIG. 8A), a push-up(FIG. 8B), and a sit-up (FIG. 8C). As demonstrated in these figures, themodel generalizes to unseen people with a negligible increase of thekeypoint localization error but has varying performance on differenttypes of unseen tasks. For example, as shown in FIGS. 8A and 8C, thelearned model easily generalizes to poses with pressure maps similar tothe pressure maps on which the model was trained. However, as shown inFIG. 8B, the learned model delivers a less accurate performance withtactile imprints that the model has never encountered. In this instance,the model failed to predict the push-up pose, which induces pressureimprints that are vastly different from the training distribution, andinstead generalized to the lunging pose, where the pressure maps aremainly directed by the human's center of mass. When deploying the systemmore generally, it is understood that a more systematic data collectionprocedure covering more typical human activities will be needed toachieve a more reliable pose estimation performance. More accuratepredictions, nevertheless, can be achieved once additional data iscollected in view of the present disclosures, and thus extending thepresent disclosures to other poses is within the scope of the presentdisclosure. A person skilled in the art will understand many other posesthat can be predicted in view of the present disclosures.

To obtain a deeper understanding of the learned features in the poseestimation network, action classification can be performed by applying alinear classifier on the downsampled tactile feature maps. In thestudies associated with the present disclosures, this was done using thedataset on one single person performing 10 different actions, where 80%was used for training, 10% for validation, and 10% for testing.

FIG. 9 shows results on action classification including a confusionmatrix of action classification using a linear classifier on the learnedfeatures from the pose estimation model (left) and representativetactile frames from different actions (right). As demonstrated in FIG.9, the prototype system obtained an accuracy of about 97.8%, suggestingthat the learned features contain semantically meaningful information onthe input tactile frames and demonstrating the capability of the modelto facilitate downstream classification tasks.

The model was extended for multi-person pose estimation. As discussedabove, the multi-person pose estimation model was trained and validatedwith 112,000 and 10,000 pairs of synthesized tactile frames and keypointconfidence maps. Performance was evaluated with 4,000 recorded tactileframes of two people performing stepping, sitting, lunging, twisting,bending, squatting, and standing on toes. FIG. 10 shows results onmulti-person scenarios including Euclidean distance between thepredicted multi-person 3D skeleton and the ground truth. The followingtable shows the average keypoint localization error of body parts alongthe X, Y, and Z axis in the real-world coordinate:

Shoul- El- An- Axis Ave. Head der bow Waist Hip Knee kle Feet X 14.514.1 10.1 15.3 24.7 10.2 12.6 14.1 14.9 Y 12.9 13.9 10.8 15.9 21.6 10.111.0 9.7 9.9 Z 12.7 16.6 13.2 17.3 23.9 10.0 8.0 6.5 6.4

FIGS. 11A-11B show some qualitative results of exemplary multi-person 3Dhuman pose estimation including, from top to bottom, the RGB image forground truth annotation, the captured tactile frame, the ground truth 3Dskeleton, and the predicted 3D skeleton from the model using only thetactile frames (unit: cm). The images in FIG. 11B are a continuation ofthe sequence shown in FIG. 11A. The network learns to localize eachindividual and predict the corresponding 3D pose. Purely from thetactile information, the network successfully localizes each individualand predicts his or her 3D pose with a localization error of less thanabout 15 cm. The predictions do not rely on any visual information and,therefore, are unaffected by visual obstructions or a limited field ofview, which are common challenges in vision-based human pose estimation.

The prototype system was necessarily limited, for example, by thelimited input datasets used to train the system in terms of both thelimited number of subjects recorded and the limited number of activitiesrecorded. As a result, the prototype system expectedly showed various“failure” cases, which actually help in demonstrating how the systemworks and how the system can be expanded with additional input trainingsequences. FIG. 12 shows typical “failure” cases encountered in theprototype system. The typical failure cases can be categorized intothree main types. First, the model fails to predict the position of thewaist and the head (FIG. 12a ). This is expected as it can be observedthat the pressure distributions of the tactile maps are rarely or notaffected by the movement of the head and wrist when a person is standingon feet. Also, the model fails to predict the poses where actions areperformed without notable physical contact with the floor, e.g.,free-floating legs during sit-ups and twisted torso during thestanding-up process (FIGS. 12b and e ). Furthermore, different actionsmay induce very similar pressure imprints, e.g., bending and twisting,causing trouble for the model to distinguish the activities due to theintrinsic ambiguity of the tactile signal (FIGS. 12c and d ). As for themulti-person pose estimation, additional errors can happen because ofthe ambiguity underlying the tactile signals from different individuals,where the model fails when two people are too close to each other.Generally speaking, these failure cases are not inherent failures of theprototype system itself but instead can be attributed in large part tothe limited content of the synthetic training dataset for the prototypesystem. It is anticipated that these “failures” could be circumvented byvirtue of processing additional datasets in view of the presentdisclosures to arrive at further successful human pose predictions.

Furthermore, even with the constraint on the human body link lengths,some predicted human poses appear unrealistic in real life. Theforegoing notwithstanding, it is anticipated that the presentdisclosures will further support improved predicated 3D pose estimationby imposing adversarial robustness as a prior to further constrain thepredicted 3D human pose.

Also, while the prototype system used the same model for bothsingle-person and multi-person pose estimation, this approach suffersfrom the ambiguity of the tactile signal induced by multiple people thatare too close to each other. To obtain more accurate predictions onmulti-person pose estimation, a region network can be applied tolocalize the tactile information belonging to each of the individuals,which will then respectively pass through the pose estimation network topredict the pose of each person. Further details about how this can beaccomplished would be understood by a person skilled in the art in viewof the present disclosures, including the materials incorporated hereinby reference.

It should be noted that once the model is trained on an appropriatedataset, 3D pose estimation can be performed dynamically based ontactile information obtained from an intelligent carpet or otherappropriate sensor system in real-time. Furthermore, 3D pose estimationsystems and methods can be configured or trained to characterize posesand correlate them with specific actions. For example, the system mightbe trained to associate a particular pose with a particular action andcould be configured to generate a signal upon detecting certain actions,e.g., hand and body motions might be used as inputs in a video gamesystem, or a pose suggestive of someone wielding a handgun might be usedby a security monitoring application (e.g., in a home, bank, store,government building, etc.) to generate an alert. Thus, for example, 3Dpose estimation systems and method of the types described herein can beused in a wide range of potential applications including, withoutlimitation, action recognition, smart homes, healthcare, and gaming, toname but a few.

Thus, 3D pose estimation systems and methods of the types describedherein can employ a low-cost, high-density, large-scale tactile sensingcarpet or other sensing system for sensing interactions between asubject and the ground and, leveraging perception results from a visionsystem as supervision, can learn to infer 3D poses using only thetactile readings of the subject interacting with the ground. Suchsystems and methods introduce a sensing modality that is different andcomplementary to vision-based systems, opening up new opportunities forpose estimation unaffected by visual obstructions and video-basedprivacy concerns in a seamless and confidential manner.

It should be noted that while various aspects are described withreference to the use of a tactile sensing floor covering, the same orsimilar concepts (e.g., recording pressure and/or other tactileinformation and training a neural information processing system based onsynchronized video or other training data) can be used with sensorsystems that can be placed on the subject in order to record thesubject's interactions with the ground, such as, for example and withoutlimitation, “wearable” devices incorporating sensor systems (e.g.,socks, footwear, footwear insoles/inserts, bandages or other medicalwraps/devices, etc., some examples of which are described in detailbelow) and sensors that can be attached to the subject or otherwiseplaced between the subject and the ground (e.g., a base or footingsincorporating sensors such as for a machine or robot).

The present disclosure also provides for textiles made from functionalfibers capable of acting as sensors. The sensors allow the textiles tobe “smart textiles.” While textiles such as garments having sensorsexist, the textiles resulting from the present disclosures fit and actto a user just as a “non-smart” textile would while providing thebenefits of a “smart textile.” This is in contrast to existing “smarttextiles,” which are typically more rigid and/or not manufacturable in ascalable way. While existing “smart textiles” typically employtechniques such as weaving and embroidery to form their textiles, thepresent disclosure employs knitting as its technique for manufacturingits “smart textiles.” Weaving interlocks its fibers in a manner suchthat the resulting textile is not stretchable or flexible in anymeaningful manner. Garments having arbitrary shapes such as gloves andsocks are not typically woven because it would be difficult to do and/orwould result in a stiff, uncomfortable, and possibly not useablegarment. A manufacturer would have to make sheets of woven materials andsew them together to create a garment like a glove or sock usingweaving. Knitting, on the other hand, creates loops that interconnect,thus allowing for three-dimensional geometries to be more easilycreated. Garments having arbitrary shapes such as gloves and socks canbe knitted. The result is garments that are flexible, wearable, and notstiff, contrary to yarn, which would be considered stiff in suchcontexts and likely could not be used with the techniques provided forin the present disclosure. Weaving needs additional tailoring ofmultiple pieces to form an actual garment, while knitting can directlyfabricate the whole garment, providing for easier fabrication ofgarments. Additionally, weaving is generally limited to flat surfacesand monotonous surface textures, while knitting allows for the conformaldesign of complex 3D geometries and versatile surface textures. Stillfurther, the knitting techniques provided are scalable in a manner thatallows such smart textiles to be mass produced using automated machinesto do the knitting, a feature not achievable using existing smarttextile-making techniques, such as embroidery.

It should be noted that sensor systems used for 3D pose estimation asdiscussed above (e.g., carpets, mats, socks, shoe insoles/inserts,bandages, flooring, etc.) can include or be fabricated with fibers ortextiles having sensors including functional fibers of the typesdescribed herein. It also should be noted that calibration techniquesdescribed herein can be applied equally to 3D pose estimation systemsand methods such as for characterizing and calibrating apressure-sensing carpet or mat.

As described herein, functional fibers that include a conductive core(e.g., a stainless steel thread) and a piezoresistive coating (e.g., apolydimethylsiloxane elastomer) disposed around a circumference of thecore are well-suited for use with the knitting techniques provided forforming garments having arbitrary shapes. Further, the combination ofthe functional fibers and the knitting techniques means that the smarttextiles can be fabricated in an automated manner, allowing for the massproduction of smart textiles that function akin to counterpart textilesthat do not include sensors or are not otherwise “smart.” As usedherein, “automated” includes being able to fabricate or otherwiseproduce an object, function, etc. without any human labor intervention.The fabricated garments can be referred to as whole-garment sensingbecause the entire garment can be fabricated from the functional fibers,meaning the whole garment can provide sensing capabilities.Alternatively, the functional fibers can be incorporated into garmentsat select locations as desired to create garments having certain areasor zones where sensing is desirable. The systems and methods providedfor herein allow for creation of garments and other textiles thatprovide a sensing platform across virtually an entire surface area ofthe garment/textile, the sensing platform being high-density,3D-conformal, and low cost.

Fiber Fabrication+Knitting

FIG. 13A illustrates a coaxial piezoresistive fiber produced using acontinuous fabrication method for low-cost (about 0.2 US dollar/m). Thecontinuous fabrication method can be an automated fiber-pulling set-up.FIGS. 13B and 13C provide for one non-limiting example in which acommercial conductive stainless steel thread, shown in FIG. 13B, iscoated with a piezoresistive nanocomposite, shown in FIGS. 13C, 13D, and13E. More specifically, FIG. 13B illustrates a morphology of each of:(a) a stainless steel thread; (b) a functional fiber, such as thestainless steel thread being coated by a piezoresistive nanocomposite;and (c) an acrylic yarn. The piezoresistive nanocomposite can include,for example, a polydimethylsiloxane (PDMS) elastomer as the matrix andgraphite/copper nanoparticles as the conductive fillers. Thecross-section of the functional fiber shown in FIG. 13C illustrates thestainless steel thread as a central portion of the fiber with the PDMSelastomer encompassing a circumference of the thread. Magnified views ofthe functional fiber are shown in FIGS. 13D and 13E, with FIG. 13E inparticular showing a uniform dispersion of nanoparticles that can beachieved, for example, after shear mixing the nanocomposite at a highrate of speed.

While the illustrated embodiment provides for a stainless steel core,the core can be any thread, filament, wire, or other configurationhaving conductive properties. Other metals and/or conductive polymerscan be used in lieu of, or in combination with, stainless steel.Likewise, while the illustrated embodiment provides for a piezoresistivecoating that includes PDMS, the coating can be any thermoset orthermoplastic that achieves similar effectiveness. For example, thecoating can be a polymer that is impregnated or otherwise filled withfillers to give it changing resistive properties with respect to someexternal signal. Further, while in the present disclosure pressurechanges are sensed and relied upon to make various determinations andpredictions, alternatively, or additionally, other properties can beused too. For example, changes in temperature, a pH level, a chemicallevel, an electro-magnetic property, an acoustic parameter, a vibration,etc. are other parameters that can be sensed, and thus the formulationof the fiber coating can be adapted in conjunction with the same. Stillfurther, multiple fibers that sense different signal types can beincluded in the same garment and/or materials, such as a carpet and thelike, described above, conducive to detecting changes in multipleproperties can be utilized for the coating.

Each sensing unit can be constructed by orthogonally overlapping twopiezoresistive fibers. FIG. 13F illustrates the electricalcharacteristic of a single sensor composed of two such coaxialpiezoresistive fibers. As shown, such a configuration can convertpressure (normal force) stimuli into electrical signals exhibiting aresistance drop (approximately in a range from about 8 kΩ to about 2 kΩ)when a force approximately in the range of about 0.05 N to about 2 N isapplied. FIG. 13G demonstrates that no decrease in performance isobserved in the functional fiber even after more than 600 load andunload cycles. The sensor characteristic is stable and reliable over atleast that usage/time frame. FIG. 13H illustrates an electricalcharacterization on devices composed of different combinations of fabricstructures, including manual inlays, automatic inlays, a combination ofautomatic and manual inlays, functional fabrics with fabrics, andfunctional fibers without fabrics. The performance (e.g., sensitivityand detection range) of the functional fiber is affected, at least inpart, by the processing parameters (e.g., pulling speed, pressure, andcoating thickness, as shown in FIG. S1 of Appendix B of the provisionalpatent application from which the present application claims priority)and materials compositions (e.g., copper and graphite weight percentage,as shown in FIG. S1 of Appendix B). The functional fiber can be stableat room temperature (up to about 50° C.), but its resistance canincrease with temperature afterward, as also shown in FIG. S1 ofAppendix B.

The formed functional fibers can then be seamlessly integrated intofabrics and full garments through programmable digital machine knitting.Due to the interlocking loops (or stitches), knitted fabric enjoysadditional softness and stretchability as compared with woven fabric. Aplurality of functional fibers knitted together as provided for hereincan be referred to as a “knit.” Furthermore, machine knitting canrealize versatile surface textures, complex 2D/3D shapes and geometries,as well as maximal conformability during wearing, enabling the scalablefabrication of full-garment wearables that are compatible with dailyhuman activities. The functional fibers can also be integrated intosmart carpets and the like, as described in greater detail above.

To accommodate the mechanical characteristics of the piezoresistivefunctional fiber, a knitting technique, inlaying, can be employed.Performed automatically on a knitting machine, inlaying horizontallyintegrates the yarn in a substantially straight configuration, whichcannot be directly knitted by forming loops due to their relativefragility and stiffness. In some embodiments, to optimize themanufacturability and device performance, two methods of inlaying can beemployed: automatic inlaying and manual inlaying. Additional informationrelated to the same can be found in Appendix B, such as the portionassociated with FIG. S4. Two knitted fabrics with functional fiberinlaid in orthogonal directions can be assembled as a sensing matrix.Referring again to FIG. 13H, and as also shown in FIG. 14, sensorsensitivity and detection range can be highly influenced by the knittedpattern. Generally, sensitivity decreases and detection range increaseswith the integration of fabrics. A sensor composed of two fabrics withautomatic inlaid functional fibers can have the lowest sensitivity andhighest detection range, for instance, because the ribbed texture cancreate a gap between two fabrics and lower the force response.

FIG. 14A illustrates one non-limiting example of a scalablemanufacturing technique that can be used to form wearable garments thatutilize functional fibers like those described above. A person skilledin the art, in view of the present disclosures, will recognize suchtechniques can also be applied to manufacturing a carpet and otherobjects provided for herein or otherwise derivable from the presentdisclosures. As shown, a stainless steel thread can be fed forward usinga fiber-feeding, or fiber-pulling, system, such as by a pulley inconjunction with a winding system, to a location where the coating isapplied. The fiber-pulling system can be customized as desired such thatit can an provide length-scale coaxial piezoresistive fiber fabricationand digital knitting processes for seamless integration of sensingmatrices in fabrics and garments. The coating in the illustratedembodiment is nanocomposites fed onto the thread. Many techniques knownto those skilled in the art for applying a coating to a fiber or threadcan be employed. The combination of the thread and the nanocompositescan then be cured. In the illustrated embodiment, the curing occursfurther down the manufacturing line and is done thermally, although inother embodiments the curing can occur at the location where the coatingis applied to the thread and/or other techniques for curing can beemployed. A person skilled in the art will appreciate other techniquesand configurations can be used to advance or otherwise move thestainless streel thread to a location where a coating will be applied. Afiber-feeding system can be any combination of components (e.g., gears,pulleys, belts, etc.) operable to advance the stainless steel thread (orother component being used to form a conductive core of a functionalfiber) in conjunction with forming the functional fiber and/ormanufacturing a textile with a functional fiber (referred to herein as atransport system). The result of the cured coating on the stainlesssteel thread is a functional fiber that includes a conductive core and apiezoresistive coating. FIG. 14A further shows a digital knittingmachine that can be used to combine a plurality of functional fiberstogether to form a wearable, such as a garment, having sensors disposedthroughout the wearable. Thus, the illustrated embodiment of FIG. 14Aprovides one non-limiting example of an inexpensive large-scalemanufacturing system and method to create tactile sensing textiles.

Many wearables (e.g., garments) can be formed in view of the presentdisclosures. Some non-limiting examples are provided in FIGS. 14B-14E.As shown, large-scale tactile sensing matrices can be embedded intofull-sized gloves (e.g., 722 sensors, FIG. 14B), socks (e.g., 672sensors, FIG. 14C), vests (e.g., 1024 sensors, FIG. 14D), and robot armsleeves (e.g., 630 sensors, FIG. 14E). A commercial digital knittingsystem, which is known to those skilled in the art, can allow thesetypes of garments to be fully customizable, thus allowing for garmentstailored based on an individual size and/or an individual preference(s)(e.g., color, particular design features), meeting the needs ofpersonalization and fashion design. Details about the knitting operationand designs can be found in Appendix B, such as the portion associatedwith FIGS. S3-S5. In at least some embodiments, a modifiedelectrical-grounding-based circuit can be used to extract signals fromeach individual sensor.

To the extent the present disclosure describes garments as beingwearable, a person skilled in the art will appreciate that othergarments or textiles that are not necessarily wearable by a human canalso be produced in accordance with the present disclosures. By way ofnon-limiting examples, the garments produced based on the disclosedsystems and methods can be placed on objects like robots (or portionsthereof), machines, furniture, vehicle seats, and/or on floors and/orwalls to sense some sort of action. By way of further non-limitingexamples, the garments produced based on the disclosed systems andmethods can be used in garments for animals, such as clothing, saddles,etc. Accordingly, the term “wearable garment” can encompass any garmentor textile that can be placed on and/or over an object, human, or animalthat allows some sort of action to be sensed.

FIG. 14F illustrates that the provided for systems and methods allow forthe collection of large tactile datasets (up to a million frames or evenmore) over versatile human-environment interactions. More particularly,FIG. 14F illustrates the data flow of the data pipeline to understandhuman activity and pose classification, showing input data fromdifferent garments going into a neural network, with the outputincluding classifications identified above. Such disclosures can also beapplied to carpets and the like, as provided for above. Each sensor canbe represented by a circle, and as pressure is applied, an indication ofwhere the pressure is being applied can appear, such as by using variouscolors and/or intensities of colors to reflect changes in resistance. Asprovided for herein, the present systems and methods, and the wearablesand other object (e.g., carpets) that are produced using the same, canbe coupled with machine learning techniques, self-supervised sensingcorrection, physical interactions identification, human behaviorsignatures discovery, and/or full-body motion predictions to provideeven further benefits from and/or for the wearables. Examples of some ofthese benefits are described in greater detail below and/or theAppendices. For instance, detailed network architectures are illustratedin FIG. S7 of Appendix B.

Self-Supervised Sensing Correction/Calibration

While researchers have attempted to fabricate flawless sensor arrays,sensor variation and failure have been inevitable during scale-up anddaily applications. In contrast, living organisms can adapt theirsensory system in the presence of individual sensor failure orvariation. The present disclosure provides for a similar mechanism thatcan relax current strict standards in sensor fabrication. Restricted byhigh-density sensing units, complex geometries, and diverse applicationscenarios, it is impractical to perform individual correction of eachsensor in the provided embodiments. Thus, a self-supervised learningparadigm is provided that learns from weak supervision, usingspatial-temporal contextual information to accommodate malfunctioningsensors and compensate for variation. More particularly, synchronizedtactile responses are collected from the garment(s) (e.g., the glove)and readings from a digital scale pressed by a wearer, as shown in FIG.15A and, as referenced in Appendix B, data S2. At each frame, the scalereading indicates the force being applied, which is expected to linearlycorrelate with the sum of tactile responses at all sensing points. Afully convolutional neural network (FCN) can be trained to take in asmall sequence of raw tactile array responses and output a single framewith the same spatial array resolution, representing the calibratedresult of the middle frame of the input sequence (as shown in FIG. S7Aof Appendix B). The neural network can be optimized via a stochasticgradient descent (SGD) with the objective having two components: one canencourage the output to preserve the details in the input and the othercan restrict the calibrated tactile response to be close to the readingfrom the scale. The network can increase the correlation between thetactile response and the reference (reading from scale). In oneexemplary embodiment, illustrated by FIG. 15A, as well as FIG. S8, A toD, of Appendix B, the correlation was increased from approximately 77.7%to approximately 88.3% for the glove, and from approximately 92.4% toapproximately 95.8% and from approximately 75.9% to approximately 91.1%for the left and right socks, respectively.

The same self-supervised learning framework can be employed using thecorrected glove as a new “scale” to process the sensing fabrics witharbitrary shapes, such as a vest and robot arm sleeve, as shown in FIG.15B and, as referenced in Appendix B, data S2). In this illustratedexemplary embodiment, the correlation was increased from approximately32.1% to approximately 74.2% for the vest and from approximately 58.3%to approximately 90.6% for the robot arm sleeve (see also FIG. S8E ofAppendix B).

The self-supervised calibration network can exploit the inductive biasunderlying the convolutional layers, can learn to remove artifacts, andcan produce more uniform and continuous responses, as supported by FIGS.15C-15F, as well as FIG. S9 of Appendix B. Further, the provided forcalibration network enables the large-scale sensing matrix to beresistant to individual variation and even disruption, and thereforeensures the quality of extracted information.

FIG. 17A illustrates an additional example of calibrating a glove. It isa snapshot from a video that illustrates the estimation of the loadcalculated form the glove readout compared to the scale readout. Asshown, the glove is again used in conjunction with a digital scale, withthe wearer of the glove applying a force to the scale at various angles,positions, etc. The data can be collected and calibrated in accordancewith the present disclosures.

As shown by FIG. 15B, the calibrated glove can become a new “scale” forcalibrating other garments, such as a vest and/or robot arm sleeve.Similarly, one or more other datasets can be collected by pressing thetarget garment using the calibrated glove, whose responses can beregarded as the “reference,” and train another calibration network usingthe same network architecture and training procedure. The tactileinformation from the calibrated glove can be collected and can reflectthe physical force being applied while pressing on the tactile vestwearing the calibrated tactile glove. In some examples, like the one inFIG. 15B, correlation between the tactile response and the “reference”increases from approximately in the range of about 32.1% to about 74.2%for the vest after self-supervised calibration. Again, similar methodscan be used with other wearables, such as the calibration of a sensingsleeve. In some instances, the correlation between the tactile responseand the “reference” for the sleeve increase from approximately in therange of about 58.3% to about 90.6% for the robot sleeve. Examples ofcalibrated tactile response from a glove, a vest, socks, and a robot armsleeve are provided in FIGS. 15C, 15D, 15E, and 15F, respectively. Ineach instance, artifacts are removed and matrix uniformity improves. Asdiscussed above with respect to FIG. 14F, each sensor of the wearablecan be represented by a circle in the raw signal and self-supervisedimages, with the sensors responding in some fashion when pressure isapplied to them. In the illustrated embodiments, locations of pressure,and more specifically changes in resistance indicative of locations ofpressure, are illustrated by different colors and/or intensities ofcolors, although other illustrations and/or indications are possiblewithout departing from the spirit of the present disclosure. Theself-supervised images result from the calibrations provided for herein.

The self-supervised calibration network can exploit the inductive biasunderlying the convolutional layers, learn to remove artifacts, andproduce more uniform and continuous responses, among other capabilities.It enables the large-scale sensing matrix to be resistant to individualvariation and even disruption and therefore can ensure the quality ofextracted information. As provided for herein, calibration can be usedto fill-in holes where data is lost or otherwise corrupted.

While the illustrated embodiment provides for a glove, any type ofcovering for a hand can be adapted in a similar manner, including butnot limited to mittens, wraps, or medical bandages. A person skilled inthe are will also understand how to apply these same principles tocarpets and the like in view of the present disclosures.

Classification+Signatures

The reliability, stability, and wearability of full-body sensinggarments coupled with self-supervised calibration pipeline as providedfor herein allows a large tactile dataset (over 1,000,000 framesrecorded at 14 Hz) on versatile human-environment interactions to becollected. Such datasets can include data related to object grasping,complex body movement, and daily locomotion. The capability of theprovided for systems and methods can be tested and demonstrated, by wayof non-limiting examples, by extracting useful information for actionidentification, motion prediction, signatures discoveries, andenvironment classification.

Vest

A full-sized sensing vest (with 1024 sensors in one non-limitingembodiment) illustrated in at least FIGS. 15D and 16A shows the exactforce distribution during sitting, standing, lying, and other actions,which can mirror a wearer's posture, activity status, and theshape/texture of the contacting object. With the increasing burden ofhealthcare, especially for the elderly and disabled, such smartwearables offer a solution as automatic health monitoring system, whichcan trigger an alarm in an emergency (e.g., sudden fall), warning,and/or provide information for early disease detection (e.g., heartattacks or Parkinson's disease). Notably, such alarms, warnings,detections can be implemented in any wearable produced in accordancewith the present disclosures, and thus are by no means limited to vests.Also, because such wearable is soft and comfortable, it can be asuitable choice for infant movement/body position tracking andidentifying potential neurodevelopmental disorders.

Furthermore, the sensing matrix provided for herein demonstrates asuperior sensitivity than a human's back. By way of example, and asshown in FIG. 16B, a dataset can be collected by pressing models ofthree letters (e.g., M, I, and T) against the back of a mannequinwearing a tactile vest of the present disclosure from differentorientations. The data can be categorized into 10 classes and a simpleneural network that takes a small window of responses as input can betrained to determine and/or predict the type of the letter and theorientations. During some testing, the classification networkdemonstrated it can achieve an accuracy of 63.76%, which can drop as theeffective resolution decreases from about 32×32 to about 1×1, asillustrated in FIG. 16. This illustrates the benefit of higherresolution as compared to a human's back.

FIG. 17D illustrates an additional example of calibrating a vest using acalibrated glove. As pressure is applied by the glove to the vest,sensors on each of the glove and the vest respond by changing colorsand/or intensifying colors based on the amount of pressure experienced.The raw and calibrated data can be provided for both the glove and vest.More particularly, the concept of self-calibration, described above,allows the sensing garment to be calibrated with arbitrary geometry.

While the illustrated embodiment provides for a vest, any type of topcan be adapted in a similar manner, including but not limited to shirts,coats, sweaters, sweatshirts, blouses, wraps, undergarments (e.g.,undershirts, some types of t-shirts, bras, lingerie), or medicalbandages. Likewise, these disclosures can also be applied to bottoms,including but not limited to pants, trousers, shorts, undergarments(e.g., underpants, long johns, lingerie), or medical bandages.Whole-garment sensing wearables can be extended into various industriesand fields, and the garments associated with the same, to provide usefulinformation for those fields, including but not limited to athletics(e.g., particular types of garments associated with different sports),construction (e.g., gear used on construction sites), medical (e.g.,medical masks), and military (e.g., uniforms worn in training orcombat). A person skilled in the art will appreciate that thesedisclosures can likewise be applied to objects outside of wearables,such as carpets and the like, as provided for herein and/or as derivablefrom the present disclosures.

Action Classification+Clustering

For example, human action identification can be achieved based ontactile information obtained from a pair of socks integrated withfunctional fibers. The dataset can be collected by the user wearing thesock and performing various daily activities, including walking forward,walking backward, side-walking, walking upstairs/hill, walkingdownstairs/hill, leaning, jumping, standing, standing on tiptoes,lifting a leg (as shown in top image of FIG. 15E), squatting, twisting,turning (as shown in bottom image of FIG. 15E), and bending over (e.g.,to touch toes), among other actions that can be performed by a wearer ofthe sock(s). The system can take in a desired number of tactile framesretrieved from the left and right sock (e.g., 45 frames), each of whichcan be passed individually through two convolutional layers eachfollowed by a rectified linear unit (ReLU) activation and maxpooling.The resulting hidden layers can be passed through a linear layerfollowed by a softmax to predict associated class of the task type. Asdiscussed above, human action identification can also be achieved by wayof a carpet or the like, in addition to or in lieu of socks or otherfootwear.

Motion Prediction

As discussed above, motion prediction can be achieved by the presentsystems and methods. Further illustrations related to the same areprovided with respect to FIGS. 16, 17B, and 17C, as well as FIG. S14 andthe referenced S5 movie from Appendix B. The systems (sometimes referredto as a network) provided for in the present disclosure are able todifferentiate patterns of footprints across different actions, and thusthe capability of tactile socks, carpets, floors, etc. can be furthertested by training a similar system (or network) to predict a human'spose.

Humans maintain the dynamic balance of the body by redirecting thecenter of mass and exerting forces on the ground, which results indistinct force distributions on the feet. A person's pose can beestimated from a change of force distribution over time obtained bytactile socks as provided for herein as a sequence of pressure maps. Forexample, the body pose can be represented by 19 joint angles spanningover the legs, torso, and arms. Synchronized tactile data from a pair ofsensing socks and a full-body motion capture (MOCAP) suit can berecorded, while the user performs versatile actions. The pose predictiontask can be modeled as a regression problem using a convolutional neuralnetwork. The model can process a time-series of tactile array footprintsthat can contain the evolving information about the contact events andcan predict the human pose in the middle frame. The neural network canbe optimized by minimizing the mean-squared error (MSE) between thepredicted and the ground truth joint angles (MOCAP data) using SGD.Further details can be found in Appendix B and the descriptions andreferences to figures below.

FIG. 16A illustrates example photographs and tactile frames that canassist in the identification of diverse sets of signatures, such asdiscussed above. FIG. 16B provides a T-SNE plot from the pose datasetfrom the tactile vest, described above. The separable clustersillustrate the discriminative capability of the sensing vest.

FIG. 16C provides example photographs and tactile frames of the letters“M,” “I,” and “T” pressed on the tactile vest for classifying the letterand the orientation. FIG. 16D provides a confusion matrix, illustratingthat accuracy drops as effective resolution decreases.

FIG. 16E is an illustration of 19 different joint angles that can bepredicted by a model as provided for herein. FIG. 16F provides for theMSE in pose production. As shown in FIGS. 16G and 16H, there can be aninfluence of sensor resolution and number of input frames (contextwindow), respectively, on prediction performance. The dashed line ineach figure represents a baseline performance where the predictions arethe canonical mean poser obtained from the training data.

FIG. 16I provides a comparison of various poses reconducted from MOCAP(ground truth) and tactile frames from the socks as provided for herein(prediction). Discrepancies in predicting the pose of the arm arecircled.

FIG. 16J provides for a time series prediction of walking in view of thepresent disclosures, and FIG. 16K provides principal component analysis(PCA) on tactile maps from walking, with the insets corresponding totactile frames.

FIGS. 17B and 17C illustrate two examples of motion prediction of a userwearing tactile socks is made based on tactile feedback from the sensorsin the socks. In FIG. 17B, the tactile feedback of the sensors isillustrated, an image of the stance of the person providing the force onthe tactile socks is shown (the “Ground Truth”), and an image of whatthe model predicts the stance of the person providing the force on thetactile socks is provided. The prediction and actual stances arestrikingly similar. FIG. 17C is presented in a similar manner, with thetactile feedback of the sensors illustrated, an image of the stance ofthe person providing the force on the tactile socks shown as the “GroundTruth,” and an image of what the model predicts the stance of the personproviding the force on the tactile socks is provided. Again, theprediction and actual stances are strikingly similar, even down to theplacement of the hands.

While the illustrated embodiment provides for socks (also referred to asstockings), any type of foot covering can be adapted in a similarmanner, including but not limited to shoes, boots, slippers, or medicalbandages. Likewise, and as described in greater detail above, thepresent disclosures allow for these determinations to be made by way ofa carpet, floor, or other similar objects.

Robot Arm

In addition to the sensing wearables described herein, the systems andmethods disclosed can also work as skin for a robot. Most modern robotsrely solely on vision; however, in the fields of robot manipulation andhuman-robot interaction, large-scale and real-time tactile feedback canbe a critical component for more dexterous interaction skills,especially when vision is occluded or disabled. The sensing wearable canenable conformal coverage on the robotic gripper, limbs, and otherfunctional parts with complex 3D geometries, endowing the robots with astrong tactile sensing capability. FIG. 15F illustrates a robot arm(sometimes referred to herein or elsewhere as “KUKA”) equipped with theconformal sensing skin of the present disclosure. The robot arm canreceive real-time tactile feedback and can feel the touch of a humanbeing. It obtains huge potential in unobtrusive multi-point collisionand interaction detection, as shown in FIG. 18, which remainschallenging with the embedded torque sensors in the robot arm andconventional computational tools. Therefore, the present systems andmethods step up as an important ingredient to facilitate futurecooperation between humans and robots (or service robots).

The sleeve can serve as a skin of the itself, or alternatively, theouter-most layer of a robot can be configured to have a textile like thesleeve as part of it to form the skin of the robot. This can allow fordesired tactile feedback for the robot, and the host of applicationsthat can result from the same.

FIG. 17E illustrates an additional example of calibrating a sleeve usinga calibrated glove. As pressure is applied by the glove to the sleeve,sensors on each of the glove and the sleeve respond by changing colorsand/or intensifying colors based on the amount of pressure experienced.The raw and calibrated data can be provided for both the glove andsleeve.

The results attributable to the present disclosures demonstrate a broadutility of the integrated platform coupling scalable manufacturing andcomputational pipeline and highlight its potential in human-environmentinteraction learning, which is an integral step toward the convergenceof human and artificial intelligence. Certain exemplary embodimentsbridge the gap between functional fibers and industrial-scale textilemanufacturing, enabling monitoring, recording, and understanding ofhuman daily behaviors and activities. The present disclosures allow fortraining data to be recorded and analyzed in a wide variety of contexts.For example, training data of baseball players with wearable tactilegloves can be recorded and analyzed for optimized training strategy.Once combined, the platform provided for by the systems and methodsherein allows full-body data collection, including systematicinformation on human movement, and diverse human-environmentinteractions, which may lead to breakthroughs in healthcare, robotics,service robots, human-computer interactions, biomechanics, education,and smart interactive homes, among other industries and uses.

While various exemplary embodiments focus on garments, any type oftextile can be fabricated, calibrated, and used in accordance with thepresent disclosures. Some non-limiting examples of such textiles includecarpet and furniture. The type of garments that can be used inconjunction with the present disclosures is essentially limitless. Asdiscussed above tops, bottoms, gloves, and socks can all be formed usingthe systems and methods provided for herein, as can other types ofgarments not explicitly described or illustrated, such as headwear(e.g., hats, caps, wraps, medical bandages), among other garments wornby humans, animals more generally, robots, or machines more generally.

Further, the present disclosure provides for sensors that enableidentifying and/or predicting human activity, but they are by no meanslimited to use with human activity. The systems and methods provided forherein can also be used in the context of a control system, such as byproviding sensor feedback to allow for parameters to be monitored and/oractions to be taken in response to the same. By way of furthernon-limiting example, the systems and methods provided for herein can beused to identify activities and/or events having to do with animals,robots, machinery, and/or in an environment.

The priority patent application, along with any descriptions and claimsprovided for herein, provide the relevant description of the variousdisclosures of the present patent application. One skilled in the artwill appreciate further features and advantages of the invention basedon the above-described embodiments and the content of the prioritypatent application. Accordingly, the invention is not to be limited bywhat has been particularly shown and described, except as indicated bythe appended claims. Features from one embodiment can typically beimplemented in other embodiments. By way of non-limiting example, afeature made possible by the functional fiber being used to form asensing wearable vest (e.g., alerts, warnings, or alarms, as discussedabove) can typically be carried over into other wearables, carpets, etc.as well. The disclosure of a feature in one embodiment by no meanslimits that feature from being incorporated into other embodimentsunless explicitly stated. All publications and references cited hereinare expressly incorporated herein by reference in their entirety,including references provided for in the priority patent application.

It should be noted that headings are used above for convenience and arenot to be construed as limiting the present invention in any way.

The disclosed systems and methods (e.g., as in any flow charts or logicflows described above) may be implemented using computer technology andmay be embodied as a computer program product for use with a computersystem. Such embodiments may include a series of computer instructionsfixed on a tangible, non-transitory medium, such as a computer readablemedium (e.g., a diskette, CD-ROM, ROM, or fixed disk). The series ofcomputer instructions can embody all or part of the functionalitypreviously described herein with respect to the system.

Those skilled in the art should appreciate that such computerinstructions can be written in a number of programming languages for usewith many computer architectures or operating systems. Furthermore, suchinstructions may be stored in any memory device, such as a tangible,non-transitory semiconductor, magnetic, optical or other memory device,and may be transmitted using any communications technology, such asoptical, infrared, RF/microwave, or other transmission technologies overany appropriate medium, e.g., wired (e.g., wire, coaxial cable, fiberoptic cable, etc.) or wireless (e.g., through air or space).

Among other ways, such a computer program product may be distributed asa removable medium with accompanying printed or electronic documentation(e.g., shrink wrapped software), preloaded with a computer system (e.g.,on system ROM or fixed disk), or distributed from a server or electronicbulletin board over the network (e.g., the Internet or World Wide Web).In fact, some embodiments may be implemented in a software-as-a-servicemodel (“SAAS”) or cloud computing model. Of course, some embodiments ofthe invention may be implemented as a combination of both software(e.g., a computer program product) and hardware. Still other embodimentsof the invention may be implemented as entirely hardware, or entirelysoftware.

Computer program logic implementing all or part of the functionalitypreviously described herein may be executed at different times on asingle processor (e.g., concurrently) or may be executed at the same ordifferent times on multiple processors and may run under a singleoperating system process/thread or under different operating systemprocesses/threads. Thus, the term “computer process” refers generally tothe execution of a set of computer program instructions regardless ofwhether different computer processes are executed on the same ordifferent processors and regardless of whether different computerprocesses run under the same operating system process/thread ordifferent operating system processes/threads. Software systems may beimplemented using various architectures such as a monolithicarchitecture or a microservices architecture.

Importantly, it should be noted that embodiments of the presentinvention may employ conventional components such as conventionalcomputers (e.g., off-the-shelf PCs, mainframes, microprocessors),conventional programmable logic devices (e.g., off-the shelf FPGAs orPLDs), or conventional hardware components (e.g., off-the-shelf ASICs ordiscrete hardware components) which, when programmed or configured toperform the non-conventional methods described herein, producenon-conventional devices or systems. Thus, there is nothing conventionalabout the inventions described herein because even when embodiments areimplemented using conventional components, the resulting devices andsystems (e.g., processing systems including neural informationprocessing systems) are necessarily non-conventional because, absentspecial programming or configuration, the conventional components do notinherently perform the described non-conventional functions.

The activities described and claimed herein provide technologicalsolutions to problems that arise squarely in the realm of technology.These solutions as a whole are not well-understood, routine, orconventional and in any case provide practical applications thattransform and improve computers and computer systems.

While various inventive embodiments have been described and illustratedherein, those of ordinary skill in the art will readily envision avariety of other means and/or structures for performing the functionand/or obtaining the results and/or one or more of the advantagesdescribed herein, and each of such variations and/or modifications isdeemed to be within the scope of the inventive embodiments describedherein. More generally, those skilled in the art will readily appreciatethat all parameters, dimensions, materials, and configurations describedherein are meant to be exemplary and that the actual parameters,dimensions, materials, and/or configurations will depend upon thespecific application or applications for which the inventive teachingsis/are used. Those skilled in the art will recognize, or be able toascertain using no more than routine experimentation, many equivalentsto the specific inventive embodiments described herein. It is,therefore, to be understood that the foregoing embodiments are presentedby way of example only and that, within the scope of the appended claimsand equivalents thereto, inventive embodiments may be practicedotherwise than as specifically described and claimed. Inventiveembodiments of the present disclosure are directed to each individualfeature, system, article, material, kit, and/or method described herein.In addition, any combination of two or more such features, systems,articles, materials, kits, and/or methods, if such features, systems,articles, materials, kits, and/or methods are not mutually inconsistent,is included within the inventive scope of the present disclosure.

Various inventive concepts may be embodied as one or more methods, ofwhich examples have been provided. The acts performed as part of themethod may be ordered in any suitable way. Accordingly, embodiments maybe constructed in which acts are performed in an order different thanillustrated, which may include performing some acts simultaneously, eventhough shown as sequential acts in illustrative embodiments.

All definitions, as defined and used herein, should be understood tocontrol over dictionary definitions, definitions in documentsincorporated by reference, and/or ordinary meanings of the definedterms.

The indefinite articles “a” and “an,” as used herein in thespecification and in the claims, unless clearly indicated to thecontrary, should be understood to mean “at least one.”

The phrase “and/or,” as used herein in the specification and in theclaims, should be understood to mean “either or both” of the elements soconjoined, i.e., elements that are conjunctively present in some casesand disjunctively present in other cases. Multiple elements listed with“and/or” should be construed in the same fashion, i.e., “one or more” ofthe elements so conjoined. Other elements may optionally be presentother than the elements specifically identified by the “and/or” clause,whether related or unrelated to those elements specifically identified.Thus, as a non-limiting example, a reference to “A and/or B”, when usedin conjunction with open-ended language such as “comprising” can refer,in one embodiment, to A only (optionally including elements other thanB); in another embodiment, to B only (optionally including elementsother than A); in yet another embodiment, to both A and B (optionallyincluding other elements); etc.

As used herein in the specification and in the claims, “or” should beunderstood to have the same meaning as “and/or” as defined above. Forexample, when separating items in a list, “or” or “and/or” shall beinterpreted as being inclusive, i.e., the inclusion of at least one, butalso including more than one, of a number or list of elements, and,optionally, additional unlisted items. Only terms clearly indicated tothe contrary, such as “only one of” or “exactly one of,” or, when usedin the claims, “consisting of,” will refer to the inclusion of exactlyone element of a number or list of elements. In general, the term “or”as used herein shall only be interpreted as indicating exclusivealternatives (i.e., “one or the other but not both”) when preceded byterms of exclusivity, such as “either,” “one of,” “only one of” or“exactly one of” “Consisting essentially of,” when used in the claims,shall have its ordinary meaning as used in the field of patent law.

As used herein in the specification and in the claims, the phrase “atleast one,” in reference to a list of one or more elements, should beunderstood to mean at least one element selected from any one or more ofthe elements in the list of elements, but not necessarily including atleast one of each and every element specifically listed within the listof elements and not excluding any combinations of elements in the listof elements. This definition also allows that elements may optionally bepresent other than the elements specifically identified within the listof elements to which the phrase “at least one” refers, whether relatedor unrelated to those elements specifically identified. Thus, as anon-limiting example, “at least one of A and B” (or, equivalently, “atleast one of A or B,” or, equivalently “at least one of A and/or B”) canrefer, in one embodiment, to at least one, optionally including morethan one, A, with no B present (and optionally including elements otherthan B); in another embodiment, to at least one, optionally includingmore than one, B, with no A present (and optionally including elementsother than A); in yet another embodiment, to at least one, optionallyincluding more than one, A, and at least one, optionally including morethan one, B (and optionally including other elements); etc.

In the claims, as well as in the specification above, all transitionalphrases such as “comprising,” “including,” “carrying,” “having,”“containing,” “involving,” “holding,” “composed of,” and the like are tobe understood to be open-ended, i.e., to mean including but not limitedto. Only the transitional phrases “consisting of” and “consistingessentially of” shall be closed or semi-closed transitional phrases,respectively, as set forth in the United States Patent Office Manual ofPatent Examining Procedures, Section 2111.03.

Although the above discussion discloses various exemplary embodiments ofthe invention, it should be apparent that those skilled in the art canmake various modifications that will achieve some of the advantages ofthe invention without departing from the true scope of the invention.Any references to the “invention” are intended to refer to exemplaryembodiments of the invention and should not be construed to refer to allembodiments of the invention unless the context otherwise requires. Thedescribed embodiments are to be considered in all respects only asillustrative and not restrictive.

What is claimed is:
 1. A system for identifying activity of a subjectrelative the ground, the system comprising: a tactile sensing floorcovering for sensing interaction of the subject with the ground; and aprocessing system in communication with the sensor system and having atleast one processor coupled to a non-transitory memory containinginstructions executable by the at least one processor to cause thesystem to: receive an input tactile sequence produced from sensorsignals generated by the tactile sensing floor covering sensor system;compare the received input tactile sequence against information in adatabase that correlates tactile information to particular activities;and identify the activity of the subject based on the comparison.
 2. Thesystem of claim 1, wherein the identified activity includes at least oneof an identified movement or an identified position of at least one partof the subject.
 3. The system of claim 1, wherein the instructionsfurther cause the system to: trigger a notification based on theidentified activity.
 4. The system of claim 3, wherein the notificationcomprises at least one of an alarm, a warning, or an indication of anearly disease detection.
 5. The system of claim 1, wherein the tactilesensing floor covering comprises at least one of a carpet, rug, mat,floor cloth, pad, plank, tile, sheet, or other flooring product.
 6. Thesystem of claim 1, wherein the tactile sensing floor covering comprises:a piezoresistive pressure sensing matrix fabricated by aligning anetwork of orthogonal conductive threads as electrodes on each side of acommercial piezoresistive film, wherein each sensor is located at theoverlap of orthogonal electrodes.
 7. The system of claim 1, wherein theinstructions further cause the system to: implement an encoder that mapsthe input tactile sequence into a 2D feature map, expands and repeatsthe 2D feature map to transform the 2D feature map into a 3D featurevolume comprising a plurality of voxels, and appends an indexing volumeindicating the height of each voxel; and implement a decoder that runsthe appended and indexed 3D feature volume through a set of decodinglayers to generate a predicted confidence map for each of a plurality ofkeypoints, wherein the predicted confidence map is used for comparingthe input tactile sequence against information in the database thatcorrelates tactile information to particular activities and identifyingthe activity of the subject based on the comparison.
 8. The system ofclaim 1, wherein the processing system comprises a neural informationprocessing system.
 9. The system of claim 1, wherein the instructionsfurther cause the system to: collect tactile information for a pluralityof test subjects along with reference information; and process thecollected tactile information and the reference information to producethe information in the database that correlates tactile information toparticular activities.
 10. The system of claim 9, further comprising: atleast one camera, wherein the reference information comprises video orimages from the at least one camera of the test subjects producing thecollected tactile information.
 11. A method for identifying activity ofa subject relative the ground, the method comprising: receiving, by aprocessing system, an input tactile sequence produced from sensorsignals generated by a tactile sensing floor covering that sensesinteraction of the subject with the ground; comparing, by the processingsystem, the received input tactile sequence against information in adatabase that correlates tactile information to particular activities;and identifying, by the processing system, the activity of the subjectbased on the comparison.
 12. The method of claim 11, wherein theidentified activity includes at least one of an identified movement oran identified position of at least one part of the subject.
 13. Themethod of claim 11, further comprising: triggering, by the processingsystem, a notification based on the identified activity.
 14. The methodof claim 13, wherein the notification comprises at least one of analarm, a warning, or an indication of an early disease detection. 15.The method of claim 11, wherein the tactile sensing floor coveringcomprises at least one of a carpet, rug, mat, floor cloth, pad, plank,tile, sheet, or other flooring product.
 16. The method of claim 11,wherein the tactile sensing floor covering comprises: a piezoresistivepressure sensing matrix fabricated by aligning a network of orthogonalconductive threads as electrodes on each side of a commercialpiezoresistive film, wherein each sensor is located at the overlap oforthogonal electrodes.
 17. The method of claim 11, further comprising:implementing, by the processing system, an encoder that maps the inputtactile sequence into a 2D feature map, expands and repeats the 2Dfeature map to transform the 2D feature map into a 3D feature volumecomprising a plurality of voxels, and append an indexing volumeindicating the height of each voxel; and implementing, by the processingsystem, a decoder that runs the appended and indexed 3D feature volumethrough a set of decoding layers to generate a predicted confidence mapfor each of a plurality of keypoints, wherein the predicted confidencemap is used for comparing the input tactile sequence against informationin the database that correlates tactile information to particularactivities and identifying the activity of the subject based on thecomparison.
 18. The method of claim 11, wherein the processing systemcomprises a neural information processing system.
 19. The method ofclaim 11, further comprising: collecting tactile information for aplurality of test subjects along with reference information; andprocessing the collected tactile information and the referenceinformation to produce the information in the database that correlatestactile information to particular activities.
 20. The method of claim19, wherein the reference information comprises video or images of thetest subjects producing the collected tactile information.